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PREFACE 


Development  of  the  procedures  reported  herein  was  sponsored  by  the 
Office,  Chief  of  Engineers,  as  part  of  the  Environmental  and  Water 
Quality  Operational  Studies  (EWQOS)  Program,  under  Task  II. D,  Reservoir 
Regulation  for  Water  Quality  Management.  The  investigation  was  con¬ 
ducted  and  the  report  prepared  during  the  period  June  1980  to  May  1981 
by  Dr.  Darrell  G.  Fontane  and  Dr.  John  W.  Labadie  of  Colorado  State 
University  and  Mr.  Bruce  Loftis  of  the  Waterways  Experiment  Station  (WES), 
Hydraulics  Laboratory.  Mr.  H.  B.  Simmons,  Chief  of  the  Hydraulics 
Laboratory,  WES;  Dr.  John  Harrison,  Chief  of  the  Environmental  Labora¬ 
tory,  WES;  Mr.  Frank  A.  Herrmann,  Assistant  Chief  of  the  Hydraulics 
Laboratory;  Mr.  John  L.  Grace,  Jr.,  Chief  of  the  Hydraulics  Structures 
Division;  Dr.  Dennis  R.  Smith,  Chief  of  the  Reservoir  Water  Quality 
Branch;  and  Dr.  Jerome  L.  Mahloch,  Program  Manager  of  EWQOS,  provided 
general  guidance. 

Director  of  WES  during  the  investigation  was  COL  N.  P.  Conover,  CE. 
Technical  Director  was  Mr.  F.  R.  Brown. 

This  report  should  be  cited  as  follows: 

Fontane,  D.  G.,  Labadie,  J.  W. ,  and  Loftis,  B. 

1981.  "Optimal  Control  of  Reservoir  Discharge 
Quality  Through  Selective  Withdrawal;  Hydraulic 
Laboratory  Investigation,"  Technical  Report 
E-82-1,  prepared  by  Colorado  State  University 
and  the  Hydraulics  Laboratory,  Waterways  Experi¬ 
ment  Station,  for  the  U.  S.  Army  Engineer  Water¬ 
ways  Experiment  Station,  CE,  Vicksburg,  Miss. 
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REFERENCES 


OPTIMAL  CONTROL  OF  RESERVOIR  DISCHARGE  QUALITY 
THROUGH  SELECTIVE  WITHDRAWAL 

Hydraulic  Laboratory  Investigation 


PART  I :  INTRODUCTION 

1.  Federal  agencies  involved  in  the  operation  of  systems  of  man¬ 
made  lakes  are  confronted  with  a  complexity  of  interdependent  decisions 
within  a  multipurpose  framework.  Federal  reservoir  projects  are  under¬ 
taken  for  such  purposes  as  flood  control,  navigation,  hydropower,  water 
supply,  recreation,  and  water  quality  control.  Of  these,  flood  control 
and  water  quality  control  present  the  greatest  challenges  for  effective 
operation.  In  management  for  these  purposes,  rapid  decisions  are  often 
required,  decisions  that  must  be  made  with  uncertainty  about  future 
conditions  and  that  can  have  a  significant  impact  on  other  project 
purposes. 


Objective 

2.  The  purpose  of  this  report  is  to  discuss  in  general  the 
management  of  density-stratified  reservoirs  for  water  quality  control 
and  to  present  a  procedure  for  determining  optimal  release  strategies 
to  meet  downstream  temperature  requirements.  The  procedure  is  based 
upon  a  combined  simulation-optimization  approach  in  which  a  reservoir 
water  quality  simulation  model  is  used  to  evaluate  alternatives  sug¬ 
gested  by  an  optimization  method.  Temperature  is  the  only  water  quality 
parameter  considered  in  the  procedure.  The  procedure  can  be  extended 

to  include  additional  water  quality  parameters,  but  it  is  difficult  to 
develop  an  objective  function  which  accurately  reflects  preferences 
between  parameters  that  sometimes  conflict. 

Reservoir  Regulation 

3.  Many  reservoirs  can  be  operated  to  release  water  of  a  specific 


3 


quality.  In  general,  the  quality  of  the  water  within  a  reservoir  varies 
with  both  time  and  space.  The  variation  is  usually  most  pronounced  dur¬ 
ing  thermal  stratification,  when  surface  water  is  warmed  and  cooler 
water  remains  near  the  bottom  (Figure  1).  The  resulting  density- 
stratified  condition  inhibits  vertical  mixing  and  affects  various 


Figure  1.  Representation  of  selective  withdrawal  structure  operation 
for  a  thermally  stratified  reservoir 

hydrodynamic  processes  within  the  reservoir.  As  a  consequence,  the 
quality  of  the  water  varies  with  Its  location  in  the  reservoir.  Further¬ 
more,  this  variation  is  generally  most  pronounced  vertically.  With 
knowledge  of  the  vertical  distribution  of  temperature  within  a  reservoir, 
a  selective  withdrawal  outlet  works  (Figure  2)  which  provides  the  flexi¬ 
bility  of  withdrawing  water  of  the  desired  quality  from  various  strata 
in  the  lake  can  be  designed. 

4.  Reservoirs  can  be  operated  to  achieve  in-lake  objectives  such 
as  evacuating  waters  with  low  dissolved  oxygen  content  from  the  bottom 
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Figure  2.  Example  of  selective  withdrawal  structure 


j  or  releasing  a  density  current  of  inflowing  suspended  sediment  resulting 

}  from  a  storm  in  the  upstream  watershed.  Most  often,  however,  reservoirs 

i 

j  are  operated  to  meet  downstream  temperature  objectives  during  the  thermal 

;  stratification  cycle.  Downstream  temperature  objectives  are  established 

i 

>  to  enhance  a  coldwater  or  warrawater  fishery  or  to  maintain  pre-project 

I  stream  temperature  conditions.  Release  temperature  can  also  be 


important  for  other  reasons  such  as  irrigation. 

5.  Operational  decisions  for  daily  control  of  release  temperature 
are  usually  based  on  current  conditions.  Thermal  simulation  models  are 
used  to  evaluate  the  capability  of  selective  withdrawal  designs  to  meet 
release  temperature  objectives  by  using  hydrologic  and  meteorologic 
data  to  simulate  the  thermal  stratification  cycle.  Algorithms  are 
included  within  the  models  to  simulate  operation  of  the  selective 
withdrawal  structure.  Since  the  algorithms  used  incorporate  only  current 
and  not  anticipated  future  conditions,  any  sequential  operational 
decisions  must  be  regarded  as  myopic.  From  consideration  of  in-lake 
temperature  conditions,  flow  requirement,  outlet  works  geometry,  and 
downstream  target  temperature,  daily  decisions  can  be  made  as  to  which 
selective  withdrawal  ports  should  be  open  and  what  flow  should  be 
released  through  each  outlet. 

6.  If  the  withdrawal  structure  is  adequately  designed  and  if  the 
reservoir  is  large  enough  that  sufficient  water  of  the  desired  tempera¬ 
ture  is  available  to  ensure  meeting  temperature  objectives  over  the 
stratification  cycle,  then  this  period-by-period  or  myopic  operation  is 
satisfactory.  However,  if  the  reservoir  can  provide  only  a  limited 
quantity  of  water  at  a  desired  thermal  level,  a  severe  deviation  from 
the  temperature  objectives  may  occur  during  the  latter  part  of  the  cycle 
(Figure  3).  The  release  temperature  may  be  undesirable  for  the  water’s 
intended  use.  Reducing  the  severity  of  the  deviations  from  the  down¬ 
stream  target  temperature  over  a  longer  period  of  time  will  reduce  the 
shock  effect  of  abrupt  temperature  changes  on  the  river  ecosystem.  In 
the  case  of  limited  capacity,  the  way  to  produce  desired  release  tempera¬ 
tures  over  the  entire  stratification  cycle  may  be  to  minimize:  (a)  the 
maximum  deviation  of  the  release  temperature  from  the  target  level, 

(b)  the  sum  of  the  absolute  deviations  (or  squared  deviations) ,  or 

(c)  the  weighted  sum  of  the  absolute  deviations  during  critical  periods 
such  as  fish  spawning. 

7.  In  this  study,  a  thermal  simulation  model  was  used  to  evaluate 
the  ability  of  selective  withdrawal  structure  designs  to  meet  release 
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OOWNSTREAM  TARGET  TEMPERATURE 
RELEASE  TEMPERATURE 


Figure  3.  Comparison  of  release  temperature  and  target 
temperature  for  a  case  study  reservoir 


temperature  objectives  at  proposed  reservoirs.  The  mathematical  model 
incorporates  hydrologic  and  meteorologic  data  to  simulate  the  thermal 
stratification  cycle  in  a  reservoir  and  includes  algorithms  to  simulate 
operation  of  each  selective  withdrawal  structure. 
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PART  II:  SOLUTION  STRATEGIES 


Static-Optimal  Release  Strategy 


8.  The  reservoir  operation  release  strategy  selected  for  water 
quality  control  is  often  identified  on  a  daily  basis,  and  the  decisions 
involved  are  made  based  only  on  the  current  state  of  the  system;  no  con¬ 
sideration  is  given  to  future  conditions.  Such  decisions  can  be  regarded 
as  myopic.  In  following  such  a  strategy,  if  decisions  are  made  such 
that  the  release  temperature  corresponds  most  nearly  to  a  downstream  tem¬ 
perature  objective,  then  the  strategy  can  be  called  static-optimal. 
Determination  of  a  static-optimal  release  strategy  is  not  a  trivial 
exercise.  The  decisions  to  be  made  involve  the  operation  of  a  multilevel 
outlet  structure  (such  as  shown  in  Figure  2)  to  release  water  with  a 
temperature  that  most  nearly  matches  downstream  temperature  requirements. 
Specifically,  the  decisions  are  which  of  the  selective  withdrawal  ports 
should  be  opened,  whether  the  larger  floodgate  should  be  opened,  and 
what  flow  should  be  released  through  each  of  the  open  outlets.  Static- 
optimal  decisions  include  the  following  considerations: 

a.  State  of  the  system;  that  is,  the  vertical  temperature 
profile  in  the  lake  on  the  day  of  interest. 

b.  Morphological  description  of  the  lake.  Because  the  volume 
varies  with  depth,  the  volume  of  water  available  for  re¬ 
lease  at  a  specified  temperature  is  a  function  of  its 
vertical  location  in  the  lake. 

£.  Total  flow  to  be  released  downstream.  Usually  this  is 

specified  by  some  other  project  purpose  such  as  hydropower 
or  flood  control. 

cl.  Downstream  target  temperature. 

e.  Hydraulic  constraints  of  the  outlet  structure.  Each  of 

the  outlets  shown  in  Figure  2  is  limited  by  a  minimum  flow 
rate,  or  rate  below  which  flow  cannot  be  controlled,  and 
a  maximum  flow  rate.  Also,  the  entire  selective  with¬ 
drawal  system  has  a  maximum  design  capacity.  And  each 
different  outlet  configuration  can  present  different  con¬ 
straints.  Sometimes  for  hydraulic  reasons  two  particular 
ports  cannot  be  operated  simultaneously;  often  it  is 
preferable  to  operate  vertically  adjacent  ports. 
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The  objective  function  for  the  static-optimal  reservoir  regulation  prob¬ 
lem  is  to  minimize  the  difference  between  the  release  temperature  and 
the  downstream  target  temperature  based  on  information  available  at 
the  time  of  interest.  If  there  is  sufficient  volume  available  in  the 
lake  and  sufficient  flexibility  provided  in  the  design  of  the  outlet 
structure,  then  a  static-optimal  release  strategy  will  result  in  release 
temperatures  which  deviate  only  slightly  from  the  downstream  target 
temperatures  for  the  entire  stratification  cycle. 

Dynamic-Optimal  Release  Strategy 

9.  For  many  projects,  a  static-optimal  release  strategy  is  en¬ 
tirely  acceptable.  However,  there  are  projects  for  which  it  is  important 
to  minimize  the  deviations  or  to  redistribute  them  in  time.  A  dynamic- 
optimal  release  strategy  requires  decisions  based  on  anticipated  future 
conditions.  Whereas  the  static-optimal  objective  is  to  minimize  the 
deviations  for  each  day  of  operation  or  simulation,  the  dynamic-optimal 
objective  is  to  minimize  some  function  of  all  of  the  temperature  devia¬ 
tions  for  all  of  the  days  in  the  simulation  period.  The  static-optimal 
problem  can  be  expressed  mathematically  as 

min 

(Tf  -  Tfc)^  for  every  n  (1) 

subject  to:  simulation  model 

where 

T^  =  predicted  release  temperature 

Tt  =  downstream  temperature  objective  or  target  temperature 

<£n  =  decision  space;  that  is,  the  combination  of  ports  that  should 
be  opened  and  the  associated  flow  rates  of  each  for  day  n 

A  typical  dynamic-optimal  problem  can  be  expressed  mathematically  as 


subject  to:  simulation  model 
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DYNAMIC-OPTIMAL  SOLUTION 


Figure  4.  Typical  system  response  to  op¬ 
timal  release  strategies 
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PART  III:  DYNAMIC  PROGRAMMING  NOTATION 


11.  Optimization  means  finding  the  best  solution  among  several 
feasible  alternatives.  Dynamic  programming  is  an  approach  to  optimiza¬ 
tion  that  takes  a  sequential  or  multistage  decision  process  containing 
many  interdependent  variables  and  reduces  it  to  a  series  of  single-stage 
problems,  each  containing  only  a  few  variables.  Reservoir  operation  is 
a  sequential  decision  process  conducive  to  solutions  by  dynamic 
programming . 

12.  Nemhauser  (1966)  has  used  five  variables  to  characterize  one 


stage  of  a  multistage  system: 


d  (decisions) 


x  (inputs) 


y  (outputs)  “  g(x,d) 


a.  An  input  state  x  that  provides  all  relevant  information 
about  inputs  to  the  stage. 

b.  An  output  state  y  that  provides  all  relevant  informa¬ 
tion  about  outputs  from  the  stage. 

£.  A  decision  variable  d  that  controls  the  operation  of 
the  stage. 

d.  A  stage  return  r  that  is  a  scalar  variable  which 
measures  the  utility  of  the  decision. 


e.  A  stage  transformation  g  that  is  a  single-valued  trans¬ 
formation  whic’-  expresses  each  component  of  the  output 
state  as  a  function  of  the  input  state  and  the  associated 
decisions;  that  is,  y  =  g(x,d)  . 


The  dynamic-optimal  reservoir  regulation  problem  can  be  formulated  as  a 
resource  allocation  problem;  that  is,  best  allocation  of  a  limited  re¬ 
source  to  each  of  many  stages.  Using  the  notation  of  Nemhauser,  the 
forward  recursive  equation  for  a  general  stage  i  for  a  simple  resource 
allocation  problem  can  be  written  as 
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subject  to:  x.^  =  x^_1  +  (4) 

for  1=1,2,  . . . ,  N 

where 

i  =  stage  index 

x  =  state  variable;  the  quantity  of  the  resource  allocated 
up  to  and  including  the  current  stage 

d  =  decision  variable;  the  quantity  of  the  resource  allo¬ 
cated  at  the  current  stage 

r  =  contribution  to  the  objective  function  at  the  current 
stage 

f  =  accumulated  contributions  to  the  objective  function,  assuming 
the  previous  decisions  were  optimal 

N  =  total  number  of  stages 

This  recursive  relation  will  serve  as  a  reference  for  notation  and  con¬ 
cepts  as  objective-space  dynamic  programming  is  presented  and  its 
application  to  reservoir  regulation  for  water  quality  management  is 
discussed  in  the  remainder  of  this  report. 
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PART  IV:  PREVIOUS  APPROACHES 


13.  The  use  of  simulation  models  in  conjunction  with  optimization 
methods  to  determine  reservoir  operation  strategies  to  satisfy  water 
quality  objectives  is  a  relatively  new  procedure  in  the  field  of  water 
resources  systems  analysis.  The  procedure  has  been  used  more  often  for 
planning  and  less  often  for  actual  operation.  For  example,  it  has  been 
used  to  determine  sizes  and  locations  of  selective  withdrawal  intakes 
(Loftis  and  Fontane  1976)  and  to  develop  improved  or  simplified  opera¬ 
tional  techniques  (Patterson  et  al.  1977;  Maynord  et  al.  1978).  Beard 
and  Willey  (1970)  developed  a  thermal  simulation  model  that  includes  a 
heuristic  procedure  to  anticipate  future  temperature  objectives  in 
determining  reservoir  operational  strategies.  Kaplan  (1974)  combined  a 
reservoir  ecosystem  simulation  model  and  a  nonlinear  optimization  tech¬ 
nique  to  determine  the  best  mode  of  operation  of  a  selective  withdrawal 
outlet  structure  considering  constraints  of  various  water  quality 
parameters.  A  scalar  index  that  commensurates  and  prioritizes  several 
water  quality  objectives  was  used  as  the  objective  function  for  this 
optimization  problem.  Kaplan's  model  solves  the  static-optimal  release 
problem  and  thus  does  not  anticipate  future  conditions. 

14.  In  research  performed  for  the  Waterways  Experiment  Station  at 
Colorado  State  University,  Farber  (1978)  combined  a  state-space  dynamic 
programming  algorithm  with  the  WESTEX  Reservoir  Heat  Budget  Model 
(Loftis  1979).  This  combination  provided  a  systematic  procedure  for 
determining  release  temperature  regulation  strategies  that  anticipated 
future  meteorological  and  hydrological  conditions.  Dynamic  programming 
was  selected  because  it  could  handle  sequential  decisions  and  system 
nonlinearities  conveniently. 

15.  Farber  formulated  the  dynamic  programming  problem  by  repre¬ 
senting  the  state  of  the  system  x^  for  simulation  period  i  as  the 
vector  of  temperatures  corresponding  to  the  various  discretized  verti¬ 
cal  layers.  The  decisions  _u  for  period  i  were  the  port  selection 
decisions;  i.e.,  which  ports  should  be  open  and  what  flow  should  be  re¬ 
leased  through  eac  .  open  port.  The  return  function  for  simulation 
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period  i  was  the  squared  difference  between  the  downstream  target 
temperature  T  ,  and  the  predicted  release  temperature  T  ,  or 

ri =  (Tt  "  Tr)2  (5) 

The  objective  function  was  then  the  sum  of  squared  deviations  for  all 
simulation  periods.  The  state  transformation  vector  function 

*!+!  =  ,1^)  (6) 

which  is  part  of  the  WESTEX  simulation  model,  provided  the  state  of  the 
system  at  the  next  time  step  based  on  the  current  state  and  current 
decisions . 

16.  Farber  used  a  proposed  reservoir  in  the  Southeast  as  a  case 
study  and  demonstrated  the  capability  of  the  combined  WESTEX-dynamic 
programming  approach  to  "save"  cold  water  during  an  early  simulation 
period  in  anticipation  of  later  needs.  It  worked  better  than  the  myopic 
static-optimal  approach,  but  the  small  size  of  the  case  study  reservoir 
limited  the  volume  of  cold  water  which  could  be  retained  for  later  use. 

17.  Farber' s  use  of  the  combined  simulation-dynamic  programming 

model  involved  some  computational  difficulties.  The  state  vector  could 
easily  have  had  a  dimension  of  30  or  more,  depending  upon  the  number  of 
vertical  layer  discretizations  necessary  to  represent  the  temperature 
profile.  In  order  to  deal  with  the  dimensionality  problem,  the  tempera¬ 
ture  profile  vector  was  represented  by  third-order  Chebyshev 

polynomials.  The  coefficients  of  the  polynomials  became  pseudo-state 
variables  because  they,  rather  than  the  actual  state  vector  (temperature 
profile),  were  manipulated.  If  a  higher  order  polynomial  is  needed  to 
adequately  describe  the  temperature  profile,  then  the  problem  can  be¬ 
come  intractable.  Although  representing  the  temperature  profile  by 
orthogonal  polynomials  reduced  the  dimensionality  of  the  problem,  the 
problem  remained  severe  because  time-consuming  simulation  was  used 
directly  in  the  computation  of  the  dynamic  programming  objective 
function. 
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18.  Recently,  new  ways  of  formulating  dynamic  programming 
problems  have  been  developed  to  deal  with  the  so-called  "curse  of 
dimensionality."  Conventional  dynamic  programming  approaches  are 
state-space  approaches;  that  is,  the  dynamic  programming  optimal 

value  or  return  function  is  defined  over  the  space  or  range  of  the  state 
variables  of  the  system.  Labadie  and  Hampton  (1979)  developed  a  dynamic 
programming  formulation  for  problems  where  the  decision  space  is  much 
more  restricted  than  the  state  space.  An  example  of  this  condition 
would  be  a  reservoir  operational  problem  where  the  maximum  controlled 
release  (the  decision  variable)  is  much  smaller  than  the  active  storage 
volume  in  the  reservoir  (the  state  variable) .  The  decision-space  ap¬ 
proach  basically  solves  the  dynamic  programming  problem  by  developing  a 
surrogate  state  variable  which  represents  the  accumulated  decisions 
from  stage  1  through  stage  i.  The  true  state  of  the  system  at  stage  i 
is  reconstructed  from  the  initial  true  state  at  stage  1  and  the  accumu¬ 
lation  of  decisions  from  stage  1  through  stage  i.  Thus,  there  may  be 
easier  ways  of  solving  the  problem  by  using  certain  surrogate  state 
variables  of  lower  dimension,  rather  than  the  actual,  high-dimensional 
state  vector.  Bertsekas  (1976)  relates  this  approach  to  the  concept  of 
a  sufficient  statistic;  that  is,  identification  of  the  minimum  amount  of 
information  needed  at  any  given  stage  of  a  sequential  decision  process 
such  that  subsequent  decisions  based  on  that  information  will  be  op¬ 
timal  or  at  least  suboptimal. 

19.  Tauxe  et  al.  (1980)  developed  an  important  extension  of  dy¬ 
namic  programming,  called  multiobjective  dynamic  programming,  to  solve 
problems  involving  multiple  noncommensurate  objectives.  The  procedure 
treats  all  objectives  other  than  the  primary  one  as  constraints.  Each 
objective  considered  as  a  constraint  is  added  as  a  state  variable  to  the 
dynamic  programming  formulation.  The  multiobjective  problem  is  rede¬ 
fined  as  a  single-objective  problem  with  multiple  state  variables.  The 
solution  to  this  problem  yields  the  value  of  the  primary  objective  as 

a  function  of  the  remaining  objectives.  The  dynamic  programming  optimal 
value  or  return  function  must  be  evaluated  over  all  original  state 
variables,  plus  the  additional  state  variables.  Though  methods  like 
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PART  V:  OBJECTIVE-SPACE  DYNAMIC  PROGRAMMING 


20.  The  problem  of  determining  optimal  operational  strategies  for 
selective  withdrawal  structures,  originally  solved  by  Farber,  can  be 
solved  over  an  objective  space  without  the  need  to  include  the  original 
state  variables  in  the  dynamic  programing  (DP)  optimal  value  function. 
This  objective-space  dynamic  programming  (OSDP)  approach  is  extremely 
powerful  since  the  original  multidimensional  problem  can  be  reduced  to 

a  one-dimensional  dynamic  programming  problem.  The  OSDP  concept  will 
now  be  explained  in  the  context  of  the  problem  of  selective  withdrawal 
structure  operation. 

21.  Suppose,  for  Instance,  that  an  operational  policy  for  a 
selective  withdrawal  structure  is  desired  such  that  the  sum  of  the 
squared  deviations  of  the  release  temperature 
perature  T  is  minimized  over  L  time  periods, 
to  minimize  S  ,  where 

L 

s  ‘  £  <Tr,l  *  Tt,l>2 
.  i“l 


from  the  target  tem- 
That  is,  the  goal  is 


(7) 


Let  the  squared  deviation  for  stage  i  be 


AD 


i 


Then,  the  sum  of  the  squared  deviations  is 


(8) 


L 

S-  J  AD1  (9) 

i-1 


Define  D^  as  the  accumulated  values  of  the  squared  deviations  from 
stage  1  through  stage  i 
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'i-1 


+  AD . 


1, 


(10) 


where  equals  zero.  Hypothetically,  plots  of  and  AD^  versus 

stage  i  would  have  the  form  shown  in  Figure  5.  The  objective  function 
can  now  be  simply  written  as:  minimize  DT  . 

-  Oj  (TRAJECTORY  RESULTING  IN  MINIMAL 

ACCUMULATED  SQUARED  DEVIATIONS) 

-  AD;  (SQUARED  DEVIATIONS  FOR  OPTIMAL 

TRAJECTORY ) 

- (EXAMPLE  NONOPTIMAL  TRAJECTORY) 


STAGE  i 

Figure  5.  Hypothetical  plots  of  and 

AD^  versus  stage  i 


22.  Notice  that  the  plot  of  D^  ,  as  generated  by  a  particular 
operational  strategy,  gives  the  appearance  of  a  state  trajectory.  If  it 
can  be  assumed  that  the  optimal  values  of  the  squared  deviations  and 
accumulated  squared  deviations,  ADJ  and  D*  ,  respectively,  are  gener¬ 
ated  by  a  unique  operational  strategy,  then  this  optimal  strategy  can  be 
determined  from  the  values  of  the  squared  deviations  at  each  stage  ADJ 
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and  the  initial  state  of  the  system  by  solving  an  inverse  problem.  Under 
this  uniqueness  assumption,  if  the  AD*  values  are  known,  it  is  possible 
to  reconstruct  the  unique  operational  strategy  that  produced  them.  As¬ 
suming  the  initial  temperature  of  the  reservoir  is  known,  a  forward- 
looking  DP  procedure  can  be  used  to  find  these  ADJ  values.  The  reason 
for  selecting  the  forward  DP  instead  of  the  more  common  backward  DP  pro¬ 
cedure  will  be  apparent  from  the  following  discussion. 

23.  The  forward  DP  problem  proceeds  as  follows.  For  stage  1,  the 
initial  state  of  the  reservoir  (the  discrete  initial  temperature  profile 
Xq)  is  known  and  a  set  of  discrete  D^  values  is  selected.  From  Equa¬ 
tion  10,  values  of  AD^  which  correspond  to  the  values  of  D^  are 
found.  Note  that  the  intent  is  to  release  water  from  the  reservoir  with 
release  temperature  T  such  that  Equation  8  is  satisfied: 


<Tr,i  -  Tt.l>' 


To  accomplish  this,  a  modified  target  temperature  T  -  is  specified 

t»1  1/2 

that  deviates  from  the  actual  target  temperature  by  (AD^)  ; 

that  is. 


Tt,i+  <4V 


(12) 


Notice  that  T  1  _>  T  .  can  be  assumed  for  this  case  since  slightly 
warmer  water  should  be  released  if  a  coldwater  objective  is  being  main¬ 
tained,  in  order  to  avoid  severe  shortages  of  cold  water  later.  For  a 
warmwater  objective  slightly  cooler  water  would  be  released. 

24.  With  T  .  specified  by  the  DP  algorithm,  the  thermal  Simula- 
*■  »* 

tion  model  is  run  for  the  first  period  using  the  initial  temperature  pro¬ 
file  or  state.  The  ports  are  regulated  to  achieve  a  release  temperature 

T  .  as  close  as  possible  to  the  modified  target  T  ,  .  If  the  simula- 
r,l  t,I 

tion  model  cannot  produce  a  release  temperature  T  -  that  achieves 

_  ^ » i  _ 

the  modified  target  temperature  T  n  ,  then  a  term  P(T  . ,T  )  is 
added  to  AD^  as  a  penalty  for  missing  the  target  temperature.  For 
stage  1,  there  is  actually  no  minimization  needed,  and  the  optimal  value 
function  is 
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w 


[1D1  +  T(Tr,l’Tt,l>] 


(13) 


where 


AD 


1 


(14) 


25.  To  summarize,  specification  of  discrete  values  yields 

AD^  which  in  turn  gives  ^  (from  Equation  12).  The  modified  target 
is  input  to  the  simulation  model  and  the  ports  are  regulated  to  achieve 
that  modified  target  as  closely  as  possible.  Any  discrepancy  is 
penalized  in  Equation  13.  The  temperature  profile  at  the  end  of  period  1 
(or  beginning  of  period  2)  is  x^(D^)  ,  which  is  also  determined  by  the 
model  and  stored  as  a  function  of  each  discrete  ;  that  is. 


=  E.1[x0»ii1(AD1)]  (15> 

where  u^(AD^)  is  the  port  operational  strategy  selected  based  on  a 
specified  target  modification  AD^  .  is  the  stage  transformation 

function  to  map  the  states  from  stage  0  to  stage  1. 

26.  In  stage  2,  the  optimal  value  function  is  defined  by 


w  - 


min  J" 

Di  L 


M>2  +  P(Tr>2,Tt>2) 


F1(D 


i>] 


(16) 


where 


AD2  =  d2  -  Dl 


Tt  ,2  -  Tt ,  2  +  (AV 


1/2 


(17) 

(18) 


and  T  „  is  obtained  from  the  thermal  simulation  model.  It  may  not  be 
r,2  _ 

possible  for  Tr  2  to  exactly  equal  Tt  2  if  the  desired  temperature 
is  simply  not  available  in  the  reservoir  or  if  the  outlet  structure  does 
not  have  sufficient  flexibility  to  release  the  desired  temperature. 
Notice  that  F2^D2^  can  written 
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w 


0,  + 


P<Tr,2'Tt,2' 


F1(D1> 


-“J 


(19) 


where 


ri(Di>  -  Di 


P(Ir.rTt,l)* 


If  it  were  possible  to  exactly  achieve  the  modified  targets  T  ^  for 
i  “  1 ,  2 ,  ....  N  ,  then 


F2(D2)  =  D2  (20) 

The  minimizing  operation  in  Equation  16  is  simply  a  means  of  finding  the 
operational  strategy  that  achieves  D2  as  closely  as  possible.  The 
reason  that  D2  is  included  in  the  right-hand  side  of  Equation  19, 
even  though  it  is  not  directly  included  in  the  minimization,  will  be 
shown  subsequently. 

27.  As  before,  the  simulation  model  determines  T  „  as  well 

r,2 

as  the  resulting  temperature  state  at  the  end  of  period  2  (or  beginning 
of  period  3) 


*2  *  £2[-l(Dl^’-2^AD2^  (21) 

which  is  stored  as  a  function  of  D2  for  the  corresponding  optimal 
;  that  is,  store  xj£(D2)  .  The  optimal  deviation  AD*(D2)  as  a 
function  of  D2  is  also  stored. 

28.  It  is  important  that  the  optima  found  in  Equation  16  are 
unique  for  each  discrete  D2  and  that  the  resultant  port  operational 
strategies  are  also  unique.  Otherwise,  there  could  be  several  possible 
end-of-period  temperature  profiles  for  the  same  release  temperature 
target.  Each  set  of  profiles  would  have  to  be  stored,  quickly  exhaust¬ 
ing  available  computer  storage  during  succeeding  stages.  The  basic 
premise  here  is  that  it  is  usually  not  difficult  to  find  a  unique  opera¬ 
tional  strategy  for  real  systems  once  an  operating  target  has  been 
specified.  There  are  generally  many  other  explicit  and  Implicit 
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objectives  that  govern  the  operation  of  the  system.  These  include  hy¬ 
draulic,  environmental,  and  even  institutional  factors,  restrictions 
(constraints)  which  actually  enhance  the  capability  of  this  algorithm. 
This  dynamic  programming  algorithm  can  regulate  the  daily  targets  to 
achieve  an  overall  optimum  for  the  entire  operation. 

29.  For  any  stage  i,  the  general  formulation  is 


w  - 


fa-  + 

Di-i  L  1 


P(Tr,i*Tt,i) 


(22) 


where 


ADi  =  Dt  - 


'i-1 


(23) 


T  ,  + 


(AD1) 


1/2 


(24) 


and  T  is  obtained  from  the  model  using  the  discrete  temperature 
profile  stored  from  stage  i-1.  The  value  of  Di  repre¬ 

sents  the  accumulated  squared  deviations  of  actual  target  temperature 
Tfc  from  the  DP-modified  target  temperature  Tt  By  definition,  F^/D^) 
is  the  sum  of  accumulated  squared  deviations  plus  minimum  total  pen¬ 
alties  from  stage  1  through  stage  i.  As  before,  the  optimal  value 

is  found  for  each  discrete  and  the  associated  optimal  AD^  (squared 

temperature  deviation  from  T  , )  for  stage  i  is  stored;  that  is, 

C  ,  X 

AD^(D*)  is  stored.  The  associated  discrete  temperature  profiles  result¬ 
ing  from  these  optimal  values  are  also  stored  for  use  in  the 

next  stage.  A  flowchart  for  the  computation  process  is  given  in  Figure  6. 

30.  Recursive  solution  of  Equation  22  will  eventually  yield  the 
function  F  (D  )  for  each  discrete  value  of  Dt  ,  where  D  is  the 
total  squar«;d  deviations  between  the  modified  target  temperature  and  the 
actual  target  temperature.  The  function  FT (D  )  is  equal  to  D  plus 
any  additional  penalties  accumulated  over  periods  1,  ....  L  due  to 
deviations  between  the  specified  modified  target  temperature  and  what 
can  actually  be  achieved.  A  hypothetical  plot  is  shown  in  Figure  7. 

The  solid  line  assumes  an  ideal  case  where  it  is  possible  to  find  a 


programming  procedure 


o 


Figure  7.  Theoretical  relationship  between 

W  and  °L 

feasible  set  of  release  temperatures  without  incurring  any  penalties. 

The  minimal  value  is  D*  .  Notice  that  values  less  than  D*  are  not 

L  L 

feasible  and  that  the  penalty  terms  therefore  dominate  FL(°L)  •  The 
function  is  linear  for  values  of  greater  than  D*  since 

F^(D^)  =  in  this  region;  that  is,  if  D*  is  feasible,  it  will 
generally  be  feasible  to  release  warmer  water — up  to  a  point,  of  course. 

31.  A  more  realistic  condition  is  shown  by  the  dotted  line  in 
Figure  7.  Here  there  is  still  some  penalty,  even  at  D*  .  Thus, 

Fl(D*)  =  D*  +  e  (25) 

If  e  is  within  a  desired  order  of  accuracy,  then  this  solution  will 
suffice.  Otherwise,  the  discretization  interval  used  for  is 

probably  too  coarse,  and  the  entire  procedure  should  be  repeated  using 
a  finer  one.  The  bounds  on  can  be  narrowed  for  this  subsequent 
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optimization,  based  on  results  from  the  coarse  discretization,  in  order 
to  stabilize  computational  cost.  Notice  that  if  the  AD^  values  were 
not  carried  in  Equation  23,  there  would  not  be  an  identifiable  minimum 
to  select:  the  plot  would  simply  be  a  nonincreasing  function.  Also, 
if  the  minimum  is  a  flat  region,  rather  than  a  distinct  minimum  point, 
there  are  no  unique  solutions.  An  arbitrary  point  could  be  selected 
but  it  would  have  to  be  regarded  as  a  poor  solution.  The  only  way  to 
guarantee  the  best  solution  would  be  to  carry  all  possible  combinations 
of  all  possible  nonunique  solutions  through  the  DP  algorithm,  which 
would  likely  be  computationally  impossible. 

32.  Once  a  specific  D*  is  selected,  a  traceback  process  is 
carried  out  to  find  the  optimal  modified  target  temperatures  T*  ,  for 
each  stage.  This  is  accomplished  using  the  stored  ADJCD^)  values. 

The  problem  takes  the  form 


Find  AD*(D*) 
then 


T?,L  *  Tt,L  +  <“t> 


1/2 


D£-i  ■  D£  -  aD£ 


How  find  AD*  (D*  ) 

and 


T*  =  T  + 

t,L-l  t,L-l 


<AD£-i> 


(26) 

(27) 


(28) 


and  so  on.  With  the  optimal  modified  target  temperatures  Tt  ±  deter¬ 
mined  for  i  =  1,  ...»  L  ,  the  simulation  model  can  now  be  run  with 
these  targets  to  produce  the  optimal  port  operational  strategies 
u* ,  i  *  1,  . « . ,  1  • 

33.  The  reason  for  using  a  forward-looking  DP  procedure  can  now 
be  seen  more  clearly.  The  forward  approach  begins  at  the  first  stage 
rather  than  the  last.  This  is  convenient  because  it  assumes  there  is  a 
specific  temperature  profile  from  which  to  start.  The  specific 


temperature  profiles  resulting  from  the  initial  profile  and  the  computed 
optimal  target  temperatures  can  be  easily  carried  forward  from  stage  to 
stage.  The  backward  approach,  on  the  other  hand,  would  impose  an  immense 
computational  burden.  At  each  intermediate  stage,  there  would  be  little 
guidance  as  to  what  starting  temperature  profile  should  be  used  since 
the  initial  stages  would  not  yet  have  been  evaluated.  The  purpose  of 
the  OSDP  approach  would  be  defeated  since  the  intermediate  stage  problem 
would  have  to  be  solved  for  all  possible  discrete  temperature  profiles 
that  could  occur. 


PART  VI:  COMPUTATIONAL  EXPERIENCE 


34.  The  OSDP  approach  was  applied  to  the  operation  of  a  reservoir 
in  northwestern  Pennsylvania.  A  generalized  one-dimensional  dynamic  pro¬ 
gramming  code  (CSU*DP)  developed  by  Labadie  and  Shafer  (1980)  was  linked 
with  the  WESTEX  simulation  model  through  a  controlling  executive  program 
that  maintained  the  integrity  of  both  models.  The  combined  WESTEX-CSU*DP 
model  was  used  to  develop  an  improved  operational  strategy  for  the  case 
study  reservoir  for  a  14-week  period.  A  comparison  between  the  squared 
deviations  of  computed  release  temperature  from  the  downstream  target 
temperature  obtained  by  the  normal  myopic  approach  of  the  simulation 
model  and  those  found  by  the  OSDP  approach  is  shown  in  Figure  8.  A  plot 


CALENDAR  DAY 

Figure  8.  Squared  deviations  of  predicted  release  temperature 
from  downstream  target  temperature  (°C) 

of  the  values  of  F^CD^)  versus  obtained  with  the  OSDP  approach  is 

shown  in  Figure  9.  For  this  case,  the  penalty  term  used  was  of  the  form 
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(29) 


Note  that  Equation  29  is  such  that  F  (D  )  actually  represents  the  rela- 

L  L 

tionship  of  Equation  7;  that  is,  the  minimum  sum  of  squared  deviations 
of  the  predicted  release  temperature  from  the  original  target  temperature. 


W 


^  <Tr,L  -  Tt,L>2 


D, 


(30) 


Figure  9.  Computed  relationship  between 
W  and  DL 

35.  The  results  shown  in  Figure  8  demonstrate  the  ability  of  the 
OSDP  approach  to  find  operational  strategies  that  consider  future  con¬ 
ditions.  The  myopic  (static-optimal)  strategy  incurs  no  deviations 
early  in  the  operation  period,  but  eventually  the  coldwater  supply  is 
depleted  and  large  deviations  occur  toward  the  end  of  the  period.  The 
operational  strategies  determined  by  the  combined  WESTEX-CSU*DP  model 
release  warmer  water  from  the  upper  level  ports  early  in  the  period, 
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thereby  incurring  some  deviations.  However,  the  coldwater  supply  under 
this  strategy  is  not  so  rapidly  exhausted,  and  the  deviations  incurred 
toward  the  end  of  the  operational  period  are  smaller. 

36.  As  discussed  in  the  formulation  of  the  objective-space  tech¬ 
nique,  the  uniqueness  of  the  D*  values  must  be  determined.  The  CSU*DP 

L 

code  contains  logic  to  "break  ties"  if  multiple  values  of  the  minimizing 
variable  yield  identical  values  of  the  optimal  return  function;  that  is, 
if  there  are  multiple  optimum  trajectories  of  the  state  variable.  The 
user  can  specify  that  either  the  first  or  the  last  tie  value  be  selected. 
The  results  shown  in  Figure  8  were  obtained  with  first-tie  selection. 

The  problem  was  then  solved  using  last-tie  selection  and  the  results 
were  identical.  Therefore,  a  unique  optimal  state  trajectory  exists, 
and  the  uniqueness  of  the  DJ  values  for  this  problem  is  strictly 
guaranteed  for  the  test  case. 

37.  Although  the  plot  of  F  (D  )  versus  D  shown  in  Figure  9 

L  L  I_i 

exhibits  the  theoretical  relationship  of  Figure  7,  the  relationship  for 
<  D*  is  not  monotonic  as  anticipated.  Conceptually,  as  the  infeasi¬ 
bility  of  the  values  increases  (D^  «  D*> ,  the  magnitude  of  the 

penalty  incurred  should  increase  accordingly.  Why  it  did  not  could  not 
be  exactly  determined  for  this  analysis.  It  is  felt,  however,  that  the 
non-monotonic  relationship  was  related  to  the  penalty  function  used 
(Equation  29)  and  that  a  different  penalty  function  or  objective  func¬ 
tion  should  produce  the  expected  monotonic  relationship. 

38.  To  evaluate  operational  strategies  for  a  different  objective 
function,  another  14-week  period  was  analyzed  by  the  combined  WESTEX- 
CSU*DP  model  with  the  objective  of  minimizing  the  maximum  deviation  that 
would  occur  during  the  period.  A  comparison  of  the  deviations  of  com¬ 
puted  release  temperature  from  the  downstream  target  temperature  obtained 
by  the  myopic  approach  and  those  found  using  the  OSDP  approach  is  shown 
in  Figure  10.  Minimizing  the  maximum  deviation  yields  an  operational 
strategy  that  incurs  deviations  throughout  the  period;  however,  the 
maximum  deviation  is  approximately  50  percent  greater  for  the  myopic 
strategy. 
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PART  VII:  CONCLUSIONS  AND  EXTENSIONS 


39.  Combining  a  thermal  simulation  model  with  a  dynamic  pro¬ 
gramming  optimization  algorithm  and  anticipation  of  future  hydrologic 
and  meteorologic  conditions  have  been  demonstrated  to  yield  improved 
operational  strategies  for  selective  withdrawal  structures.  The 
objective-space  dynamic  programming  approach  reduces  a  large  multi¬ 
dimensional  problem  to  its  equivalent  in  one  dimension  and  therefore 
eliminates  many  computational  difficulties.  Also,  the  resulting  one¬ 
dimensional  DP  problem  is  much  more  amenable  to  a  stochastic  DP 
approach  which  recognizes  that  forecasted  information  is  uncertain  and 
which  could  be  used  to  evaluate  the  effects  of  forecast  lead  time 
upon  accuracy. 

40.  Because  of  the  nature  of  the  objective-space  approach,  the 
combined  WESTEX-CSU*DP  model  could  be  formulated  such  that  the  basic 
structures  of  both  the  WESTEX  simulation  model  and  the  CSU*DP  optimizing 
model  need  not  be  altered.  The  combined  WESTEX-CSU*DP  model  can  be  used 
to  determine  operational  strategies  for  a  wide  range  of  temperature  ob¬ 
jectives,  such  as  minimizing  the  sum  of  absolute  deviations  or  minimizing 
the  sum  of  weighted  deviations.  The  approach  is  not  limited  to  a  par¬ 
ticular  reservoir  simulation  model.  Theoretically,  any  reservoir  thermal 
simulation  model  may  be  used  if  it  employs  a  multilevel  selective  with¬ 
drawal  algorithm  to  choose  the  port  operations  required  to  meet  daily 
objectives.  Additionally,  while  the  problem  presented  herein  focused  on 
the  need  to  save  cold  water  in  a  reservoir,  the  approach  could  just  as 
easily  have  been  used  for  maintaining  a  warmwater  release. 

41.  Finally,  the  method  could  be  extended  to  evaluate  other  water 
quality  parameters  in  addition  to  release  temperature.  The  objectives 
for  these  other  parameters  could  be  treated  as  constraints  in  the  one¬ 
dimensional  dynamic  programming  problem;  for  example,  minimizing  the 
sum  of  squared  temperature  deviations  subject  to  maintaining  the  dis¬ 
solved  oxygen  content  of  the  release  above  some  specified  limit.  Al¬ 
ternately,  other  water  quality  parameter  objectives  could  be  handled 
directly  by  defining  an  additional  state  variable  for  each  parameter  and 
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solving  the  prob3em  over  multidimensional  objective  space. 

42.  It  is  believed  that  there  are  many  possibilities  for  applica¬ 
tion  of  objective-space  dynamic  programming  within  the  field  of  water 
resources  systems  analysis,  in  particular,  problemr  which  have  defied 
solution  because  of  their  dimensionality. 
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